The Green Button Project

Physician Symposium Session #3, February 11 2019

Alison Callahan MISt PhD, Research Scientist, Stanford University

Alison Callahan, PhD, has no real or apparent conflicts of interest to

report.

Conflict of interest

• The Green Button origin story

• Anatomy of the consult service

• Methods and challenges

• Learning from the first 100 consults

• Deploying the service at a new site

• Ongoing efforts

Agenda

• Define use cases for an informatics consult service

• Describe requirements for setting up an informatics consult

service

• Plan deployment of an informatics consult service at a new site

Learning objectives

Informatics Consult team

Acknowledgements

Funding: NLM, NIGMS, Stanford School of Medicine, Department of Medicine, Department of

Biomedical Data Science, Center for Population Health Sciences, an anonymous donor

Stanford Health Care partners

David Entwistle

Tip Kim

Christopher Sharp

Nigam Shah

Saurabh Gombar

Robert Harrington

Alison Callahan

Vladimir Polony

Rob Tibshirani

Ken Jung

Trevor Hastie

A teenager with systemic lupus

erythematosus

• proteinuria

• antiphospholipid antibodies

• pancreatitis

Meet Laura

Source: Mayo Foundation for Medical Education and Research

Managing Laura’s care

Source: Mayo Foundation for Medical Education and Research

The Origin of the Green Button

Finding patients with

“X”

2-3 weeks to

generate a cohort

Given a specific case, provide a report with a descriptive summary

of similar patients in Stanford’s clinical data warehouse, the

common treatment choices made, and the observed outcomes after

specific treatment choices.

An institutional review board approved study (IRB # 39709) over

one year.

The Informatics Consult Service

http://greenbutton.stanford.edu

An example report

The process

Requesting

physician

Informatics

physician

EHR data

specialist

Data

scientist

Request

consult

Refine

clinical

question

Create

definitions

for

exposures

and

outcomes

Build

patient

cohorts

Perform

statistical

analysis

Write

consult

report

Review

results

Apply

evidence

to clinical

decision

24 to 72 hours

• Building patient cohorts accurately and quickly

• Asking the right question

• Controlling for confounding

• Ensuring quick turnaround

Methods and challenges

1 0 -1

Building patient cohorts

1. How will you handle time?

2. What features will you use?

3. How will you state your phenotype definition?

From timelines to data frames

Phenotyping

 Persons



 Features 

1 0 -1

Procedures

Devices

Diseases

Drugs

• cost

• utility

The search engine

• Diagnosis and procedure

codes

• Clinical notes

• Lab results

• Vital signs

• Inpatient and outpatient visits

www.tinyurl.com/search-ehr

Asking the right question

• Identify subsets of patient cohorts that are “similar”

– Matching on age, gender, record length, year

– Using propensity score matching

Controlling for confounding

• Use negative controls for empirical calibration

• E-values to quantify the degree of confounding that can produce

the observed effect

• Ask the question using multiple datasets

• Schedule an in-person debrief

What we do to not be wrong

• Search engine API available in CRAN

• R library for data pre-processing

• Semi-automated pipeline for survival and causal analyses, report

generation

Ensuring quick turnaround

Guideline

available?

Use level A

guideline

Yes

Use

Green

Button

Large cohort of patients

present?

Yes

Use

professional

judgement

Analysis + Report

• The question as posed

• How we asked the question

• Our interpretation

• Research walkthrough

List of clinical

situations

Candidates for further

study

Point of care randomization/

large simple trial

Useful

byproduct

High

priority

Increment

priority

Learning from the first 100 consults

• How many? 55%

• Which treatment? 30%

• How long? 15%

Learning from the first 100 consults

“likely to recommend” was 100%

Learning from the first 100 consults

• Institutional support

• Data science expertise

• Marketing

• A process to sanity-check

the data and consult findings

Deploying the service at your site

We’re not the first to provide an on-

demand informatics consult service

Stanford: 3 million

Optum: 55 million

Truven: 126

million

Now versus then

• What is really useful?

– Description of what happened

– Estimation: Population or Individual level

– Patient level prediction

• Financial viability – who can pay for this “test”?

• Informatics research

– Phenotyping (how do I know the patient had X)

– Representation learning

– Matching, and population level inference

– Personalized effect estimates

• Deploying as a hospital-side service at Stanford Health Care

Open questions and ongoing efforts

http://greenbutton.stanford.edu

Thank you! Questions?

@clssfr

acallaha@stanford.edu

Please complete the online session evaluation!